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Objective evaluation of audio quality is important in many fields: 

■ audio codecs 

■ effects 

■ hardware 


But... 

■ what is good audio quality? 

■ is there any objective evaluation method? 

■ how to measure audio quality? 
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Audio quality evaluation 

General approach 
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Processed audio 


Reference 



Evaluation 


i 


Score 


■ processed audio and 
unprocessed reference as 
input 

■ resulting score should be 
proportional to human 
perception 

■ mapping between numerical 
score and perceived quality 
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Audio quality evaluation 

Simple physical measures 


■ signal to noise ratio 

■ time and frequency response 

■ total harmonic distortion 

■ mean square error to reference signal 
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Audio quality evaluation 

Simple physical measures 
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Relative mean sqare error over Bitrate for MP3 
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Audio quality evaluation 

Simple physical measures 
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■ signal to noise ratio 

■ time and frequency response 

■ total harmonic distortion 

■ mean square error to reference signal 


@ simple and efficient calculation 

@ well suited for automatic evaluations with huge test sets 
Q bad correlation with human perception 
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Audio quality evaluation 

Perceptually motivated measures 
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■ new compound measures that try to mimic human hearing 

■ consider psycho-acoustic and biological aspects 

- masking effect 

- frequency-dependent hearing threshold 

- noise-to-mask ratio 

■ established tools for various applications PEAQ (audio codecs), 
PESQ (speech), PEASS (source separation) 





Relative mean sqare error [%] 


Audio quality evaluation 

Perceptually motivated measures 
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PEAQ ODG score over Bitrate for MP3 
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ODG score 








Audio quality evaluation 

Perceptually motivated measures 
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■ new compound measures that try to mimic human hearing 

■ consider psycho-acoustic and biological aspects 

- masking effect 

- frequency-dependent hearing threshold 

- noise-to-mask ratio 

■ established tools for various applications PEAQ (audio codecs), 
PESQ (speech), PEASS (source separation) 


@ usually good correlation with human perception 
@ well suited for automatic evaluations with huge test sets 
Q need for additional subjective listening tests for verification 
Q complex implementation 
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Audio quality evaluation 

Subjective evaluation methods 
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■ "listening tests" 

■ present processed signals and unprocessed references to a listener 

■ listener should rate the similarity or audio quality of the signals 


perfect correlation with human perception 
very time-consuming 
only for small test sets 
not reproducable 
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Outline 


Audio q"'’' :y evalun^ion 
* Listening test methodology 
The Be;; ;TJS framewoiK 
® Demo 

Advanced sage 
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Listening test standards 
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Humans are rarely objective in their Judgements 

it is difficult to retrieve objective results from a subjective 
evaluation 

■ use standardised test methods 

■ ITU provides several helpful publications 

- ITU-R Recommendation BS.1284-1, general methodology 

- ITU-R Recommendation BS.1116-1, ABC tests 

- ITU-R Recommendation BS.1534-1, MUSHRA tests 

■ a sufficiently big number of participants will reduce the risk of 
statistical outliers and errors 
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Listening test methodology 

Layout 
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Mushra Demo Test 


Castanets (2 of 2) 


Reference Play stop 


Test Item 1 Play 

Test Item 2 Play 

Test Item 3 Play 

Test Item 4 Play 

Test Item 5 Play 


stop 

stop 

stop 


Poor Fair Good Excellent 


Previous Test 


■ items should appear 
in random order 

■ neutral names 

■ controls for 
start/pause of 
playback 

■ possibility to loop 
short parts 
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Listening test methodology 

Audio samples 
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■ use items that underline various aspects of the tested algorithms 

■ items should be short (< 10s) 

■ overall time for test should not exceed 15 min 

■ SQAM CD for example provides a wide variety of test signals 
(speech, instruments, synthetic signals, ...) 
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Listening test methodology 

Participants 
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■ background and experience of participants can influence the 
results 

■ recommended number of participants by the ITU: 

- at least 10 expert listeners 

- more than 20 non-expert listeners 

■ use of anchor signals and hidden references to check validity 

■ well-defined listening conditions (headphones, ...) 
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Listening test methodology 

Summary 
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■ test design is crucial 

■ many pitfalls on the way to significant results 


■ difficult to reach an adequate number of participants 

■ finally, how to create a listening test?!? 
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■ test design is crucial 

■ many pitfalls on the way to significant results 


■ difficult to reach an adequate number of participants 
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^ BeaqIeJS 
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Audio q ; -‘ity evaluation 
■ Li^tf---ng netiodology 
• The BeaqIeJS framework 
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BeaqIeJS 

Listening test framework 
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■ based on HTML5 and JavaScript 

■ no dependencies to additional plugins like Adobe Flash 

■ use any modern web browser as a runtime environment 

■ two pre-defined test schemes: ABX and MUSHRA 

■ flexible usage scenarios 

- simple world-wide distribution of large-scale tests 

- distribute it locally on your workgroup server in the intranet 

- supervised tests on single computers, tablets, ... 

- perform self-blind tests 
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<!-- basic usage --> 

<audio sr c= "audio.mp3" /> 

<!-- advanced usage -*> 

<audio controls autoplay preload= "auto" > 

<source src =" audio.mp3" type= ’audio/mpeg’> 

<source src=" audio.ogg" type =’audio/ogg’> 

<p>HTML5 audio is not supported by your browser! </p> 

</audio > 

■ complete JavaScript API for playO, pauseO, loadO and 
various callbacks 

■ browser can show basic player controls 

■ create own controls with HTML and Javascript 
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Codec Support 
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Internet Explorer 

Fi refox 

Chrome 

Opera 

Safari 

WAV PCM 

no 

> 3.5 

yes 

> 11.00 

> 3.1 

Ogg Vorbis 

no 

> 3.5 

yes 

> 10.50 

with XiphQT 

MP3 

> 9.0 

> 26, not OS X 

yes 

> 14 

> 3.1 

AAC 

> 9.0 

> 26, not OS X 

yes 

> 14 

> 3.1 


■ codec support in the HTML5 <audio> element is browser 
dependant 

■ no lossless codec (e.g. FLAG) support with any browser 

■ best choice is WAV PCM with 16 bit, 44.1 kHz 
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BeaqIeJS 

Block diagram 


Local Browser, index.html 

HTML framework and DOM 
<head> 

<link href=''css/styles.css" ... /> 
<script src="config.js" ... /> 

<script src="js/jOuery.js" ... /> 
<script src="js/beaqle.js" ... /> 
</head 

<div id="container''> ... </div> 

<div id="playbackControls"> ... </div> 



Web-Service 

PHP. Python. 

Data collection 
and evaluation 
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■ Standard HTML page 

■ no server-side 
dependencies 

■ includes all necessary files 
in the header 

■ defines a set of container 
<div>-tags 

■ contents and visibility are 
controlled by the 
beaqle. js script 
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■ ListeningTest base 
class 

■ general functionality 

- test sequence control 

- playback controls 

- data loading 
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■ Audiopool class 

■ manages loading of test 
items into <audio>-tags 

■ synchronized playback 

■ callback to main class 
when audio is ready 
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■ test specific functionality 

■ layout and arrangement of 
test items 

■ evaluation and storage of 
results 
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■ interface to server-side 
scripts 

■ transmit results for 
centralised evaluation 
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ABX Listening Test 
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ABX Demo Test 


Schubert (1 of 2) 

stop Press buttons to start/stop playback. 

Please select the item which is closest to X! 


Previous Test 
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MUSHRA Listening Test 

ITU-R Recommendation BS.1534-1 
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Mushra Demo Test 



Previous Test Next Test 


00:03 


Kraft, Zolzer — BeaqIeJS: Framework for the Subjective Evaluation of Audio Quality 18 / 22 




























Outline 


Audio q ; -‘ity evaluation 
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Advanced usage 

Online submission 
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■ pack results into a JSON structure 

■ transmit them to a web service (AJAX HTTP request) 

■ store and analyse the results in a central place 


Future work: 

■ web interface to create and manage listening tests 

■ tools to evaluate and visualise results 

■ direct export to e.g. MATLAB, R, Python, ... 
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Advanced usage 

Modification/implementation of test schemes 
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function MyTest(TestData) { 

ListeningTest.apply (this , arguments); 

} 

MyTest.prototype = new ListeningTest(); 

MyTest.prototype.constructor = MyTest; 

MyTest.prototype.createTestDOM = ... 

MyTest.prototype.saveRatings = ... 

MyTest.prototype.readRatings = ... 

MyTest.prototype.formatResults = ... 

■ create a new class MyTest and inherit from ListeningTest 

■ implement test specific functionality 

— createTestDOMO 

— saveRatings0 

— readRatings0 

— formatResults0 
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Summary 

BeaqIeJS 
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■ run listening tests in any web browser 

■ flexible distribution to the participants 

■ easy to setup 

■ two pre-defined test schemes (ABX, MUSHRA) 

■ simple expansion 

■ collect and evaluate results using a central web service 

■ it is available for free 
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Where can I get it? 



https://github.com/ 
HSU-ANT/beaqlejs 
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■ code and demos are available 
on GitHub 

■ GPLv3 licence 

■ feedback and contributions 
are appreciated! 
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